Systems and methods for controlling a device via gestures received by a remote device

ABSTRACT

A method and system for sharing a user interface of a first device with a second device and enabling a user of the second device to interact with the user interface via gestures received by the second device. The first device (e.g., a smartphone) can host an application and generate a graphical user interface, which it transmits to the second device (e.g., a tablet computer) for display by the second device. The second device can receive input from a user, such as a touch input via a touchscreen of the second device, and transmit a representation of the input to the first device for providing input to the application hosted by the first device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/639,971 filed on Mar. 5, 2015, which claims the benefit of andpriority to U.S. Provisional Patent Application No. 61/948,505 filed onMar. 5, 2014, both of which are incorporated herein by reference intheir entireties.

BACKGROUND

Remote desktop systems allow a personal computer's desktop environmentto be run remotely on one system, while being displayed on a separateclient device. Some systems additionally provide for remote control ofthe personal computer by the separate client device. Many softwareapplications similarly provide for remote control of an applicationenvironment hosted by a remote computer. Using these systems, a remoteuser can interact with a computer system as if the user were physicallyin front of it.

Conventional systems that provide for remote control of a device workwell on laptop and personal computer systems, but they do not workbetween most mobile devices. Indeed, to provide remote services, thesesystems typically transmit display data from a host computer to a clientcomputer, and mouse clicks and keyboard input from the client computerto the host computer. But mobile devices typically provide only a touchscreen interface. Accordingly, conventional remote desktop systemscannot be deployed in many mobile devices.

The need exists for a system that overcomes the above problems, as wellas one that provides additional benefits. Overall, the examples hereinof some prior or related systems and their associated limitations areintended to be illustrative and not exclusive. Other limitations ofexisting or prior systems will become apparent to those of skill in theart upon reading the following Detailed Description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a suitable environment in which a system forproviding for control of a first device via input received by a remotedevice operates.

FIG. 2 is a block diagram of a host device that may be controlled bygestures received by a remote device, or a remote device that mayreceive gestures to control a host device.

FIG. 3A is a block diagram of a system for providing for control of afirst device via input received by a remote device.

FIG. 3B is a block diagram of a system according to a particularimplementation for providing for control of a first device via inputreceived by a remote device.

FIG. 4 is a flow diagram depicting a method performed by a system fordisplaying a graphical user interface generated by a host device on aremote device and controlling a first device via input received by aremote device.

FIG. 5 is a flow diagram depicting a method for processing input at aremote device for control of a host device.

FIG. 6 is a perspective view of a client media layer.

FIG. 7 is a diagram depicting interaction between components of asoftware development kit (SDK) for conveying a graphical user interfaceof a host device to a remote device and receiving input from a user ofthe remote device.

FIG. 8 is diagram of high-level steps involved in gesture control of afirst device by a second device.

FIG. 9 shows a diagram of a remote device viewing a graphical userinterface of a host device and receiving gesture input to control thehost device.

FIG. 10 shows an example graphical user interface including interfaceelements transmitted between a host device and a remote device.

DETAILED DESCRIPTION

A method and system are described for sharing, with a remote device, agraphical user interface that is generated and hosted by a host device,and providing control of the host device via input received by theremote device. Inputs received by the remote device may be in the formof gestures, i.e., touch inputs to a touch-sensitive device. The systemprocesses gestures received by the remote device to account for anydifferences in how the graphical user interface is displayed on the hostand remote device, for example due to differences in screen size, deviceorientation, and whether the graphical user interface takes up theentirety of the screen. The system may be used to facilitate, forexample, video conferencing and screen sharing between two devices. Insome embodiments, the system provides for a peer-to-peer (p2p) transferof user interface and gesture control data between the host and remotedevices. In some embodiments, the system provides forhardware-accelerated video encoding and decoding of the transmitted datato enable streaming without expensive hardware costs.

The system can commence screen sharing and gesture control when a firstdevice sends a video stream of a graphical user interface to a seconddevice. The graphical user interface may correspond to an application oroperating system (OS) running on the first device. The second devicereceives the video stream of the first device's user interface anddisplays the graphical user interface to a user of the second device.The second device receives an input gesture from the user of the seconddevice, responding to the video stream of the first device's userinterface as if it were the user interface of an application or OSrunning locally on the second device. The gesture is processed andtransmitted to the first device, and the first device implements thegesture as if it was received locally.

Various implementations of the invention will now be described. Thefollowing description provides specific details for a thoroughunderstanding and an enabling description of these implementations. Oneskilled in the art will understand, however, that the invention may bepracticed without many of these details. Additionally, some well-knownstructures or functions may not be shown or described in detail, so asto avoid unnecessarily obscuring the relevant description of the variousimplementations. The terminology used in the description presented belowis intended to be interpreted in its broadest reasonable manner, eventhough it is being used in conjunction with a detailed description ofcertain specific implementations of the invention.

The following discussion includes examples of a screen sharing andremote gesture control system used within an application for trackingfinancial accounting events, such as a banking application. Whiledepicted in a financial application, however, it will be appreciatedthat the system may be implemented within any application or environmentfor which screen sharing and remote gesture control is desired.

Suitable Environments

FIG. 1 and the following discussion provide a brief, general descriptionof a suitable computing environment 100 in which a screen sharing andremote gesture control system, as described herein, can be implemented.Although not required, aspects and implementations of the invention willbe described in the general context of computer-executable instructions,such as routines executed by a general-purpose computer, a personalcomputer, a server, or other computing system. The invention can also beembodied in a special purpose computer or data processor that isspecifically programmed, configured, or constructed to perform one ormore of the computer-executable instructions explained in detail herein.Indeed, the terms “computer” and “computing device,” as used generallyherein, refer to devices that have a processor and non-transitorymemory, like any of the above devices, as well as any data processor orany device capable of communicating with a network. Data processorsinclude programmable general-purpose or special-purpose microprocessors,programmable controllers, application-specific integrated circuits(ASICs), programmable logic devices (PLDs), or the like, or acombination of such devices. Computer-executable instructions may bestored in memory, such as random access memory (RAM), read-only memory(ROM), flash memory, or the like, or a combination of such components.Computer-executable instructions may also be stored in one or morestorage devices, such as magnetic or optical-based disks, flash memorydevices, or any other type of non-volatile storage medium ornon-transitory medium for data. Computer-executable instructions mayinclude one or more program modules, which include routines, programs,objects, components, data structures, and so on that perform particulartasks or implement particular abstract data types.

The system and method can also be practiced in distributed computingenvironments, where tasks or modules are performed by remote processingdevices, which are linked through a communications network 160, such asa Local Area Network (“LAN”), Wide Area Network (“WAN”), or theInternet. In a distributed computing environment, program modules orsubroutines may be located in both local and remote memory storagedevices. Aspects of the invention described herein may be stored ordistributed on tangible, non-transitory computer-readable media,including magnetic and optically readable and removable computer discs,stored in firmware in chips (e.g., EEPROM chips). Alternatively, aspectsof the invention may be distributed electronically over the Internet orover other networks (including wireless networks). Those skilled in therelevant art will recognize that portions of the invention may reside ona server computer, while corresponding portions reside on a clientcomputer. Data structures and transmission of data particular to aspectsof the invention are also encompassed within the scope of the invention,

Referring to the example of FIG. 1, a system operates in or among mobiledevices 105, personal computers 110, and one or more server computers115. The mobile devices 105 and personal computers 110 communicatethrough one or more wired or wireless networks 160 with each other andwith the server 115. A data storage area 120 contains data utilized bythe system, and, in some implementations, software necessary to performfunctions of the system.

The mobile devices 105 may communicate with each other and/or with thepersonal computer 110 via a peer-to-peer (p2p) connection. These devicesmay also communicate with each other through the server 115, such asthrough an application hosted at the server computer.

Suitable Devices

FIG. 2 is a block diagram illustrating a computing device 200, includinghardware components, for implementing the disclosed technology. Forexample, the device can be implemented as mobile device 105. The device200 includes one or more input devices 220 that provide input to the CPU(processor) 210, notifying it of actions performed by a user, such as atap or gesture. The actions are typically mediated by a hardwarecontroller that interprets the signals received from the input deviceand communicates the information to the CPU 210 using a knowncommunication protocol. Input devices 220 include, for example, acapacitive touchscreen, a resistive touchscreen, a surface wavetouchscreen, a surface capacitance touchscreen, a projected touchscreen,a mutual capacitance touchscreen, a self-capacitance sensor, an infraredtouchscreen, an infrared acrylic projection touchscreen, an opticalimaging touchscreen, a touchpad that uses capacitive sensing orconductance sensing, or the like.

The CPU may be a single processing unit or multiple processing units ina device or distributed across multiple devices. Similarly, the CPU 210communicates with a hardware controller for a display 230 on which textand graphics are displayed. One example of a display 230 is a display ofthe touchscreen that provides graphical and textual visual feedback to auser. In some implementations, the display includes the input device aspart of the display, such as when the input device is a touchscreen. Insome implementations, the display is separate from the input device. Forexample, a touchpad (or trackpad) may be used as the input device 220,and a separate or standalone display device that is distinct from theinput device 220 may be used as the display 230. Examples of standalonedisplay devices are: an LCD display screen, an LED display screen, aprojected display (such as a heads-up display device), and so on.Optionally, a speaker 240 is also coupled to the processor so that anyappropriate auditory signals can be passed on to the user. In someimplementations, device 200 includes a microphone 241 and camera 242that are also coupled to the processor so that spoken and video inputcan be received from the user.

The processor 210 has access to a memory 250, which may include acombination of temporary and/or permanent storage, and both read-onlyand writable memory (random access memory or RAM), read-only memory(ROM), writable non-volatile memory, such as flash memory, hard drives,floppy disks, and so forth. The memory 250 includes program memory 260that contains all programs and software, such as an operating system261, and any other application programs 263. The memory 250 alsoincludes data memory 270 that includes any configuration data, settings,user options and preferences that may be needed by the program memory260, or any element of the device 200. In some implementations, thememory also includes dynamic template databases to whichuser/application runtime can add customized templates. Theruntime-created dynamic databases can be stored in persistent storageand loaded at a later time.

As mentioned above, the device 200 also includes a communication devicecapable of communicating wirelessly with a base station or access pointusing a wireless mobile telephone standard, such as the Global Systemfor Mobile Communications (GSM), Long Term Evolution (LTE), IEEE 802.11,or another wireless standard. The communication device may alsocommunicate with another device or a server through a network using, forexample, TCP/IP protocols. For example, device 200 may utilize thecommunication device to offload some processing operations to the server115.

Device 200 may include a variety of computer-readable media, e.g., amagnetic storage device, flash drive, RAM, ROM, tape drive, disk, CD, orDVD. Computer-readable media can be any available storage media andinclude both volatile and nonvolatile media and removable andnon-removable media.

As mentioned above, the disclosed technology is operational withnumerous other general purpose or special purpose computing systemenvironments or configurations. Examples of well-known computingsystems, environments, and/or configurations that may be suitable foruse with the technology include, but are not limited to, personalcomputers, handheld or laptop devices, cellular telephones, tabletdevices, multiprocessor systems, microprocessor-based systems,programmable consumer electronics, network PCs, minicomputers, mainframecomputers, gaming consoles, televisions, e-readers, kiosk machines,wearable computers, speech generating devices, other devices for thedisabled, and distributed computing environments that include any of theabove systems or devices, and the like.

It is to be understood that the logic illustrated in each of thefollowing block diagrams and flow diagrams may be altered in a varietyof ways. For example, the order of the logic may be rearranged,sub-steps may be performed in parallel, illustrated logic may beomitted, other logic may be included, etc.

Suitable Systems

FIG. 3A is a block diagram of a system 300 for transmitting a graphicaluser interface generated by a first device to a second device, and forreceiving input at the second device for controlling the first device.The system includes a first device component 302 and a second devicecomponent 310. The first and second device components may be implementedin first and second touchscreen devices, and may be integrated intoother applications running on the first and second devices or may be astandalone application on the first and second device. For example, thefirst device component may be integrated into a mobile bankingapplication running on the first device, while the second devicecomponent may be integrated into customer support software used bycustomer support representatives of a bank. The first device component302 includes a graphical user interface component 304, a remote inputprocessing component 305, a communication component 306, and aninterface element component 308. The first device component 302 readsdata from and stores data in interface elements data storage 320. Thesecond device component 310 includes an input component 312, acommunication component 314, a display component 316, and an inputprocessing component 318.

The graphical user interface component 304 captures graphical userinterface data for transmission to the second device component. In someimplementations, the graphical user interface component generates a livestream of the graphical user interface generated by the device hostingthe first device component. For example, the graphical user interfacecomponent may encode display data of the first device, such as fromvideo memory of the first device, into a video format, such asH.264/MPEG-4. In some implementations, the graphical user interfacecomponent captures a graphical user interface at a first time fortransmission to the second device component 310 and captures thegraphical user interface at subsequent times after the graphical userinterface has been refreshed. The communication component 306 isconfigured to transmit captured graphical user interface data to thesecond device component 310 as a video stream and to receive input datafrom the second device component.

The remote input processing component 305 processes input data receivedfrom the second device component 310. In some implementations, theremote input processing component processes raw input data. However, theremote input processing component may also process input data that hasbeen pre-processed by the second device component. For example, thesecond device component may normalize input data and transmit thenormalized input data to the first device component 305, and the remoteinput processing component may convert the normalized input data to aproper format for providing input to the first device. For example, theremote input processing component 305 can convert gesture data receivedfrom the second device component to a format of the operating system ofthe first device. Based on an analysis of the user interface on thefirst device, discussed below, the remote input processing componentidentifies interface elements that are implicated by the received input.For example, the remote input processing component 305 may comparecoordinates of a received touch to areas of the graphical user interfaceencompassed by analyzed interface elements. The remote input processingcomponent notifies the first device of implicated interface elements,such that the first device executes processes or functions associatedwith the implicated interface elements as a result of the received userinput.

The interface element component 308 analyzes graphical user interfacedata and identifies interface elements in the graphical user interface.In some implementations, the interface element component 308 parses aninterface hierarchy of the graphical user interface to identify andcatalog interface elements. In some implementations, designatedinterface elements may be omitted from the catalog of interfaceelements. In some implementations, designated interface elements may becataloged but with an indication that the interface elements are not tobe implicated by received user input. The interface element componentstores data related to interface elements in the interface element datastorage area 320.

The input component 312 of the second device component 310 receives userinput from a user of the second device. User input includes a gesture,such as a touch gesture received via a touchscreen. The communicationcomponent 314 of the second device component 310 receives graphical userinterface data from the first device component 302 and transmits inputdata to the first device component. The display component 316 displaysthe graphical user interface, such as by rendering a receivedH.264/MPEG-4 video stream. For example, the display component maydisplay the graphical user interface via a touchscreen. In someimplementations, the display component resizes the graphical userinterface to fit the second device display. The graphical user interfacemay be resized to fill the entirety of or less than the full size of thedisplay of the second device display.

The input processing component 318 processes input received from a uservia the input component 312. In some implementations, the inputprocessing component 318 is configured to normalize received input basedon the aspect ratio of the graphical user interface and/or the firstdevice display. In some implementations, the input processing component318 does not process input data received by the input component, andinstead transmits raw input data and associated metadata to the firstdevice component 302.

FIG. 3B is a block diagram of a system 350 for providing screen sharingand remote gesture control of a graphical user interface according to aparticular implementation. The system includes a first mobile devicecomponent 352 associated with a first mobile device and a second mobiledevice component 360 associated with a second mobile device. In someimplementations, the system is implemented in a software development kit(SDK), which includes client applications that operate on each of thefirst mobile device and the second mobile device. For example, thesystem 350 represents a perspective view of architecture of the softwaredevelopment kit. The first device component 352 includes an API module354, a media module 356, and a gesture support module 358, and thesecond device component 360 includes a session control module (SIP) 362,a transmission module (RTP) 364, an RTCKit interface module 366, and avideo component (Doubango) 368. Doubango is an open source project forvarious technologies including 3GPP, TISPAN, Packet Cabel, WiMax, GSMA,RCS-e, IETF, audio/video coding, cloud computing, VoIP, and VNC stacks.The medial layer is discussed in more detail below.

Suitable Processes

FIG. 4 is a flow diagram representing a process 400 performed by thesystem 300 for sharing a user interface of a first device with a seconddevice and enabling a user of the second device to interact with thefirst device user interface via gestures received by the second device.For example, the first device (e.g., a smartphone) can host anapplication and generate a graphical user interface, which it transmitsto the second device (e.g., a tablet computer) for display by the seconddevice. The second device can receive input from a user, such as a touchinput via a touchscreen of the second device and transmit an indicationof the input to the first device for providing input to the applicationhosted by the first device. FIG. 9 shows a graphical representation of afirst device 905 (a tablet) being controlled by input received at asecond device 910 (a phone).

The process begins at a block 405 when the first device generates agraphical user interface. The graphical user interface may be part of auser interface for an operating system, an application operated with theoperating system, or the like. At a block 410, the first devicetransmits an indication to the second device that a graphical userinterface is available to view and interact with. The first device maytransmit the indication to the second device at the request of a user ofthe first device. At a block 415, the second device transmits a requestto the first device to transmit the graphical user interface to thesecond device. In some implementations, a user of the second devicerequests to view and interact with the graphical user interface of thefirst device prior to the first device providing any indication of itsavailability. In some implementations, the first device and the seconddevice establish a p2p connection.

At a block 420, the first device transmits the graphical user interfaceto the second device. In some implementations, the first device encodesa video stream of display data of the first device. Display data canrepresent the graphical user interface of the entire display of thefirst device or a portion of the display, such as an interface of anapplication which appears on less than the entire display of the firstdevice.

At a block 425, the first device catalogs interface elements of thegraphical user interface. Interface elements correspond to controls andother elements of the graphical user interface. Interface elements maybe associated with images or other graphics. The first device catalogsinterface elements so that when it receives input from the seconddevice, it may quickly identify if the received input is intended tointeract with an interface element.

An interface element can be active, meaning a user can interact with it,or inactive. For example, a background image on a webpage may be aninactive interface element and a link displayed over the backgroundimage may be active. Interface elements are associated with a locationon the graphical user interface. For example, interface elements can beassociated with a two-dimensional location or an area on a display, suchas a pixel location or a location relative to other elements or thedisplay size. Interface elements can also be associated with a locationof a lateral position of the interface element in a stack of interfaceelements displayed atop each other in the graphical user interface.Active interface elements are further associated with an action thatshould take place in response to an interaction with the interfaceelement.

The system may identify and catalog interface elements in differentways. In some implementations, the system parses an interface hierarchyof the graphical user interface of the first device to identify andcatalog interface elements. For example, at runtime, user interfaceelements may be represented by objects stored in memory, and the systemcan identify the user interface elements by parsing the objects found inmemory. In some implementations, the user interface is represented by aseries of UIView objects, which draw content on a display and containsubviews, creating a user interface hierarchy from which the system mayidentify interface elements.

The system catalogs interface elements at various times. In someimplementations, the system catalogs interface elements after a refreshevent. A refresh event includes each time the first device changes thegraphical user interface. For example, a refresh event may include achange in graphical user interface occurring after the first devicereceives an input from a user via a touchscreen interface of the firstdevice, or after the first device receives an input from a user of asecond device. A refresh event also includes when an application that isto be remotely controlled is opened. In some implementations, interfaceelements are cataloged each time a display is updated.

The system can store cataloged interface elements in different ways. Insome implementations, the system stores interface elements in a treestructure. The tree structure is a hierarchical representation ofinterface elements of a graphical user interface. Each node of the treeincludes information about an interface element and references tochildren interface elements. The tree may be generated from parsing theuser interface hierarchy maintained by the first device. In someimplementations, only active interface elements that can be controlledor affected by user input are added to the tree structure. In someimplementations, a data structure and application programming interface(API) handles all insertions into the tree. When an input is receivedfrom the second device, an implicated interface element may beidentified by, for example, traversing the tree structure.

In other implementations, rather than catalog interface elements in atree, the system represents the graphical user interface of the firstdevice as a matrix data structure and stores keys in all matrixlocations representing an area associated with an interface element.Subsequently, when input is received from a second device, the systemidentifies an implicated interface element based on the key stored inthe matrix at the location corresponding to the received input and thevalue associated with that key in a dictionary. For example, the systemmay store in a dictionary a representation of each pixel that is part ofan interface element, such as a UIView object, where a key correspondsto a pixel location and a value corresponds to the UIView object. Abutton, for example, may cover pixels (0,0) at its top left through(100,100) at its bottom right, and all pixels within that span of pixelsare stored with their coordinate as the key and the UIView objectcorresponding to the button as their value.

In some implementations, the system catalogs interface elements andselectively allows access to only certain of the cataloged interfaceelements. For example, an interface element may be cataloged andspecially flagged to indicate it is not enabled for remote gesturesharing by the system. Only those interface elements that have beencataloged and not flagged otherwise can be controlled or affected byinput received from the second device. Interface elements that shouldnot be enabled for remote gesture control may be specified, for example,in the configuration of the first mobile device component. An advantageof cataloging interface elements and only allowing access to certaininterface elements is that a user of the second device can still viewthe entire graphical user interface of the first device, but will beprevented from interacting with those interface elements that they arenot authorized to interact with. By doing so, the sharing of thegraphical user interface of the first device may be done more securely,by not allowing gesture controls from the second device to interact withhighly-sensitive controls on the first device. For example, in afinancial embodiment used by a bank, certain controls to enable thetransfer of money to third parties might be prevented from access by auser of the second device.

At a block 430, the first device receives input from the second device.In some implementations, the input from the second device is a singletouch coordinate. In some implementations, input from the second deviceis represented by a gesture, which may include a beginning coordinate,an end coordinate, and movement data. Movement data may indicate whethera gesture has begun, whether it has ended, whether it is moving, orwhether it was canceled (i.e., interrupted by an external event, such asan incoming phone call), and may for example be encoded in phase data ofan iOS UITouch object. The input may include other information relatedto a gesture depending on the embodiment. For example, inimplementations in which sensed motion is identified as a gesture, themotion may be represented by a general location, a direction of thegesture, and a time interval of the gesture. In some implementations,the received input is converted to an appropriate format for receivinginput at the first device. For example, if a first device utilizes anAndroid operating system, user input into a second device that istransmitted to the first device may be converted to input understood bythe Android operating system. Alternatively, the second device couldconvert the input to Android-compatible input prior to transmitting theinput.

At a decision block 435, the first device determines whether thereceived input applies to an interface element. In some implementations,the received input is compared to cataloged data associated withinterface elements to determine whether an interface element has beenimplicated by the input. In some implementations, an API traverses atree structure in pre-order fashion and identifies an interface elementthat bounds a coordinate associated with the received input. If thefirst device determines that the received input does not apply to aninterface element, the process proceeds to a decision block 437, wherethe system determines whether any changes have been made to thegraphical user interface displayed on the first device. If no changeshave been made, processing continues to block 430 where the next inputfrom the second device is received and interpreted. If changes have beenmade to the GUI displayed on the first device, however, processingreturns to block 425 where the system re-catalogs the first devicegraphical user elements and then re-interprets the input.

If at decision block 435 the first device determines that the receivedinput does apply to an interface element, the process 400 continues to ablock 440, and the first device performs an action associated with thereceived input to the interface element. The mobile device may useplatform specific method calls to perform the behavior or functionassociated with the interface element. For example, in a drawingapplication, a received input may correspond to a pressing of a pen, andthe mobile device may correlate a gesture made by a remote user across adrawing area of an application as an input of a line drawn across adrawing area of an application.

The process then proceeds to a decision block 445, where the firstdevice determines whether to terminate the stream of the graphical userinterface to the second device and the second device's ability tocontrol the first device. In some implementations, the first devicereceives an instruction from the second device to terminate theconnection between the first device and the second device. In someimplementations, the first device receives an instruction from a localuser to terminate the connection between the first device and the seconddevice. In some implementations, the system may generate an instructionto terminate the streaming when, for example, a certain period elapsesin which there is no action by either user. If the first devicedetermines that the streaming should continue, the process returns to ablock 425 where the first device catalogs interface elements. If theinstruction indicates that the streaming should cease, however, then theprocess 400 terminates.

FIG. 5 is a flow diagram representing a process 500 performed by thesystem 300 for converting input received by the second device into acompatible format for input to the first device. At a block 505, thesystem receives an input, T, from a user of the second device inrelation to a graphical user interface received from the first device,as that graphical user interface is displayed on the second device.Various characteristics of the input T are captured by the system, suchas the (x,y) coordinates of the input. The coordinates are typicallymeasured from a reference location of the displayed graphical userinterface, such as the upper left corner of the displayed interface. Ifthe second device is a touchscreen device, other characteristics of theinput may be measured such as a duration of a touch or a pressure of atouch. The process 500 may occur prior to block 430 in the process 400described above. The system may capture the input using an applicationoperating on the mobile device.

At a block 510, the system normalizes the user input to the bounds ofthe graphical user interface based on the area, V, of the graphical userinterface as displayed on the second device. The graphical userinterface area V is of size V_(x) and V_(y) in the x- and y-dimensions,respectively. The system normalizes the (x,y) coordinates of the userinput to the x- and y-dimension bounds of the graphical user interfaceas follows:

$T_{x,y}^{\prime} = {{2 \cdot \frac{T_{x,y}}{V_{x,y}}} - 1}$

At a block 515, the system accounts for an aspect ratio P₁ of thegraphical user interface displayed on the first device. In someimplementations, the system gathers information indicative of the aspectratio P₁ from sequence parameter set (SPS) information of an inboundh.264 stream of the graphical user interface. Let S represent a2-dimensional transformation matrix, the system accounts for the aspectratio as follows:

${{{If}\mspace{14mu} P_{1}} > {\frac{V_{x}}{V_{y}}\mspace{14mu} {then}}},{S = {\begin{bmatrix}1 & 0 \\0 & \frac{P_{1} \times V_{y}}{V_{x}}\end{bmatrix}\mspace{14mu} {else}}},{S = \begin{bmatrix}\frac{P_{1} \times V_{x}}{V_{y}} & 0 \\0 & 1\end{bmatrix}}$

At a block 520, the system scales the normalized received user input,T′, for transmission to the first device using the calculatedtransformation matrix S. In some implementations, T″ is clamped to [−1 .. . 1] and scaled to the range of [0 . . . 1] for transmission, asfollows:

$T_{x,y}^{''} = {\frac{\max ( {{\min ( {{T_{x,y}^{\prime} \times S},1} )},{- 1}} )}{2} + \frac{1}{2}}$

At a block 525, the system transmits a normalized and scaledrepresentation of the input T″ to the first device. In someimplementations, the input T″ is transmitted via real-time transferprotocol (RTP) to the first device. The first device multiplies thereceived representation of the user input T″ by a height and width ofthe corresponding user interface of the first device to interpret thereceived input as a corresponding input to the first device. Forexample, let Q represent a touch coordinate on the first device, and letR represent the remote client screen dimensions (assuming the userinterface fills the complete screen on the first device), then

Q=R×T″.

FIG. 6 is a perspective view of a client media layer overview, includingan output component 602 and playback component 620. In this diagram, themedial layer overview depicts an output block diagram and a playbackblock diagram. In the output component block diagram, a User InterfaceKit (UIKit) capture module 604 and an OpenGL ES capture module 606 areinputs to a screen composition module 608. OpenGL ES is a cross-platformAPI for full-function 2D and 3D graphics on embedded systems, includingconsoles, phones, appliances and vehicles. Depending on the environment,either OpenGL ES 1.X or OpenGL ES 2.X may be implemented by the system.The camera capture module 610, which captures video input from a cameraof a device, and screen composition module 608, which captures theoutput of the composed display of the device, are inputs to the H.264Encode module 612. H.264 is a next-generation video compression format.Video encoded in H.264 is transmitted from the first device to thesecond device where it is received by playback component 620. Playbackcomponent 620 decodes and displays the received video stream in afashion known to one skilled in the art. By turning the graphic userinterface on the first device into a video stream, the system allows auser of the second device to be able to view actions taken by a user ofthe first device with respect to the graphic user interface.

In some embodiments, in addition to the transmission of video from thefirst device to the second device, the output component 602 and playbackcomponent 620 also enable the transmission of audio from one device userfor another. To enable the transmission of audio, the output componentincludes a G.711 encoder module and the playback component includes aG.711 decoder module. In this fashion, audio from the first device mayalso be conveyed to the second device.

FIG. 7 is a diagram of an implementation of the system as embodied in anSDK, and a description of steps for utilizing the SDK to provide for thedisplay of a user interface on a second device and receive gesturecontrol of the interface by the second device. The SDK may be used by ahost application to enable screen sharing and remote gesture control ofthe host application. At a first step 701 the host applicationinstantiates an OpenGL ES Context provided by the SDK, instead of thedefault OpenGL ES Context provided natively, such as by the iOS orAndroid operating systems. Instantiating an SDK OpenGL ES Contextenables the screen sharing of OpenGL ES content. At a step 702, the SDKOpenGL ES Context creates a texture backed by shared memory, andattaches it to the default framebuffer object's color attachment. At astep 703, all OpenGL ES content is rendered directly to the SDK's sharedmemory buffer, without interfering with the host application. At a step704, the SDK renders this content to the original renderbuffer createdby the host application, such that it is displayed on the screen of thedevice on which the host application is running. At a step 705, framesare submitted to the SDKs screen sharing session, so that they may becomposited with UIKit content, such as interface elements provided byiOS, if applicable. At a step 706, UIKit content to be composited iscaptured, such as with the CALayer renderInContext function in the iOSoperating system. At a step 707, the UIKit view hierarchy is parsed todetermine where OpenGL ES content should be layered. At a step 708, theUIKit and OpenGL ES content are composited in a background OpenGL ESContext.

FIG. 8 shows an overview of steps involved in gesture control of a firstdevice by a second device. At a step 805, Device A receives localgestures from a user of Device A with respect to a graphical userinterface of Device B. At a block 810, Device A calculates a gesturetransformation based on aspects of Device A and Device B, such as theaspect ratio and dimensions of the graphical user interface on each ofthe two devices, and converts the local gesture to (x,y) coordinates forDevice B. Such transformation is described in additional detail in FIG.5 herein. In some implementations, rather than Device A computing thegesture transformation, Device A transmits sufficient information aboutthe touch and Device A characteristics to allow Device B to convert thetouch using an appropriate transformation. Gesture data is transmittedover an RTP connection between Device A and Device B. At a block 815,Device B traverses a user interface hierarchy for interface elementswhere the received gesture is within an interface element's bounds. At ablock 820, the client application on Device B sharing the graphical userinterface executes the gesture as if it were made locally at Device B.

FIG. 9 shows a diagram of a remote device (a smartphone 910) viewing agraphical user interface of a host device (a tablet 905) and providinginput to the graphical user interface via the remote device. Forexample, user input of an element of the GUI received on the smartphone910 causes a corresponding manipulation of the element on the graphicaluser interface on the tablet 905.

FIG. 10 shows an example graphical user interface 1100 includinginterface elements. Interface elements include buttons 1102, a textfield 1104, special buttons 1106 and 1108, and a slider 1110. As wasdescribed herein, the system catalogs interface elements and mayselectively allow access to only certain of the cataloged interfaceelements. For example, the buttons 1102 in the graphical user interface1100 may be disabled for remote gesture sharing by the system. Whendisabled, touches by a user on the displayed buttons 1102 on a seconddevice would not result in a corresponding press of the buttons 1102 onthe interface of the first device. The user of the first device wouldstill be able to depress buttons 1102 in order to execute the linkedfunctionality. All other depicted interface elements would be active forremote gesture sharing, however, meaning that the user of the seconddevice could interact with, for example, slider 1106 and text field1104. In this fashion, the application developer or owner is allowed toselectively decide which functionality is made available for remotegesture sharing.

Conclusion

Those skilled in the art will appreciate that the actual implementationof a data storage area may take a variety of forms, and the phrase “datastorage area” is used herein in the generic sense to refer to any areathat allows data to be stored in a structured and accessible fashionusing such applications or constructs as databases, tables, linkedlists, arrays, and so on.

The above Detailed Description of examples of the invention is notintended to be exhaustive or to limit the invention to the precise formdisclosed above. While specific examples for the invention are describedabove for illustrative purposes, various equivalent modifications arepossible within the scope of the invention, as those skilled in therelevant art will recognize. For example, while processes or blocks arepresented in a given order, alternative implementations may performroutines having steps, or employ systems having blocks, in a differentorder, and some processes or blocks may be deleted, moved, added,subdivided, combined, and/or modified to provide alternativecombinations or subcombinations. Each of these processes or blocks maybe implemented in a variety of different ways. Also, while processes orblocks are at times shown as being performed in series, these processesor blocks may instead be performed or implemented in parallel, or may beperformed at different times.

In general, the terms used in the following claims should not beconstrued to limit the invention to the specific examples disclosed inthe specification, unless the above Detailed Description sectionexplicitly defines such terms. Accordingly, the actual scope of theinvention encompasses not only the disclosed examples, but also allequivalent ways of practicing or implementing the invention under theclaims.

I/We claim:
 1. A method performed by a first computing device, themethod comprising: cataloging interface elements of a graphical userinterface displayed on the first computing device; transmitting video ofthe graphical user interface to a second computing device, the videocausing the display of the graphical user interface on the secondcomputing device; receiving a representation of a gesture of a user onthe second computing device, wherein: the gesture is with respect to thegraphical user interface displayed on the second computing device, andthe representation of the gesture is transformed to reflect a differencebetween the graphical user interface displayed on the second computingdevice and the graphical user interface displayed on the first computingdevice; comparing the received gesture representation to the catalogedinterface elements; identifying an interface element of the catalogedinterface elements that is implicated by the received gesturerepresentation, wherein the identified interface element is associatedwith a function; and executing the function associated with theidentified interface element on the first computing device.